o 
o 

< 



H 



SYMMETRIC SUBGROUP ACTIONS ON ISOTROPIC 

GRASSMANNIANS 

HONGYU HE AND HUAJUN HUANG 

Abstract. Let G be a classical group preserving a sesquilinear form on a vector 
space V over R or C. Let GrG(r) be the Grassmannian of isotropic r-dimensional 
subspaces. Let H ~ (Gi, G2) be a symmetric subgroup of G. In this paper, we give a 
parametrization of i7-orbits on GrG(r) in terms of dimensions of various subspaces. 
The main result of this paper is the determination of the H homogeneous structure 
and the dimension of each orbit. Consequently, we find all the open orbits. We also 
treat i7-orbits of GrG'(r) for symplectic and orthogonal groups over an algebraic 
closed field with characteristic not equal to 2. 
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1. Introduction 

The symmetric subgroup orbits in flag manifolds have been intensively studied 
in the past. Their parametrization, in the most general form, is due to Matsuki 
p!Ol [TTl [T2I [13] and Springer [16] . There are finitely many such orbits. In addition, 
there is also a natural topological ordering among the symmetric subgroup orbits, 
namely, if an orbit O' is contained in the Zariski closure of another orbit O, O' is 
said to be smaller than O. This ordering defines a partial ordering, often called the 
Bruhat ordering. The Bruhat ordering can be described purely algebraically in terms 
of the Matsuki-Springer parameter pT| [HI [151 E]- 

In this paper, we are interested in symmetric subgroup action on the Grassmannian 
of isotropic subspaces. Consider the following symmetric pairs. 

Table 1. Symmetric Pairs {G,H) and Representation Spaces V 



G 


H 


V 


Conditions 


0{p,q) 


0{pi,qi) X 0{p-pi,q-qi) 


KPi+gi 0i^P-Pi+g-gi 


< pi < p, < qi < q 


\J{p,q) 


U(pi,gi) X U{p-pi,q- qi) 


i£Pi+qi i£p-pi+q-qi 


< pi < p, < qi < q 


0„(k) 


Orra(k) X On— m(k) 


]^n-m 


< m < n 


SP2n(k) 


SP2m(k) X Sp2„_2m(k) 


J^2m ]^2ri— 2m 


< 2m < 2n 



Note: k is an algebraic closed field and char(k) 7^ 2. 



^This paper is partially supported by an NSF grant. 
^Keywords: Isotropic Grassmannian, symmetric subgroup 
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In Table [H G is a classical group that preserves a standard sesquilinear form on the 
standard representation space V, and if is a symmetric subgroup of G that stabilizes 
two subspaces U and W such that V = U Q) W. Let Grdr) be the Grassmannian 
of isotropic subspaces of dimension r. Let P be a maximal parabolic subgroup such 
that quotient G/P = Grcir). 

The first result of this paper is a parametrization of the cosets H\GiG{r) by a finite 
convex subset of a lattice. 

Theorem 1.1. Let k be an algebraically closed field with char(k) 7^ 2. Let {G,H,V) 
be given as in TableUl If G is 0„(k) or Sp2„(k), then H-orbits on GrG'(r) can be 
parametrized by an integral 4-tuple {vu, vw, a, b); If G = 0{p, q) or G = U(p, q), then 
H-orbits on Grdr) can be parametrized by an integral 5-tuple {ru,rw,(i,au,aw)- 

See Theorems 12 . 11 [STT] 16 . II and 17.11 for the definitions of parameters vu, vw, a, b, au, aw 
and the precise statements. The results for [H fl G'o)-orbit for 0{p,q) are given in 
Section 4. 

Our view point is purely algebraic. We derive our theorem by analyzing the simulta- 
neous isometry of a set of subspaces as presented in [Tf Theorem 5.3]. It is unclear 
how our parametrization should be identified with the Matsuki-Springer parametriza- 
tion [H [ini [m [16]. We shall point out that the symmetric subgroup orbits we treat 
in this paper are quite special. All the parameters can be directly determined by any 
isotropic subspace in the orbit. They sit in a positive lattice. We expect that the 
Bruhat ordering coincides with a natual ordering on the lattice. 

Another focus of this paper is to study how the group H acts on each orbit. Clearly 
each orbit can be written as H/Hs where Hs is the stabilizer of an isotropic sub- 
space S. The second result of this paper, is a determination of the stabilizer Hs- 
In the most general form, Hs is contained in a product of two parabolic subgroups 
Puixu, ru + a) oi H\u and Pw{rw, rw + a) oi H\w respectively. 

Theorem 1.2. Let G, H be as in TableUl Let S be an isotropic subspace in the 
H -orbit Ocniru, fw-, o-, *)■ Then Hs C Pu{ru, ru + a) x PiY{riv, rw + a). Moreover, 
{91192) £ Hs if and only if gi and g2 satisfy certain matching conditions. The dimen- 
sion of OcHifUi'^^WiO'-,*) is a quadratic function on the 4-tuple or 5-tuple. The open 
orbits are determined in Theorems \3. 31 \5.4l \ 6.3l \ 7.5\ 

See Theorems 13.11 \5.'2\ 16. 2| 17.21 for the precise description of the stabilizer Hs- 

Let J^G be the complete flag variety of G, and J-'q any partial flag variety of G- 
In [TT], Matsuki defined the canonical surjection 



/ : H\J^G ^ H\rai 
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then determined the inverse image f^^{0) for any iT-orbit O in J^q. For {G, H) given 
in Table [H there exists a similar canonical surjection from H\T'q to H\GiG{f) for 
every r G DIM(jF^), where DIM(jFg) is the dimension set of an isotropic flag in J^'q. 
Clearly, the following canonical map tt is injective: 

(1) ^ :f'^^ W Gra(r). 
We define a canonical map 

(2) J--H\:F'a^ n H\GiG{r). 

We show in Example 16.61 that / is not necessarily injective. So the if-orbit of a flag 
in JF^ may not be uniquely determined by the if-orbits of subspaces in this flag. 
Example 16.61 also indicates that / is not necessarily surjective. 

We shall elaborate a little bit on the motivation of this paper. Recall that func- 
tions on isotropic Grassmannian GrG'(r) can be used to define certain degenerate 
principal series Ip{v). The representation Ip{v) is one of the most intensively studied 
series of representations. In case G/P is the Lagrangian Grassmannian and G the 
symplectic group, a preliminary investigation by the first author, gives a branching 
law for the unitary Ip{v)\h (|H [S])- This branching law is multiplicity free and yields 
a Howe type L^-correspondence ( [S], ^) between unitary representations of Gi and 
unitary representations of G2 ■ So the remaining question is to see if the degenerate 
principal series in other cases will decompose in a similar fashion when restricted to 
H. A first step is to understand how H acts on GrG'(r), in particular how H acts on 
the open orbits in GrG'(r). This is what is done in this paper. We hope to discuss the 
branching law of the degenerate principal series with respect to in a future paper. 

2. Real Orthogonal Case: Orbits on Isotropic Grassmannians 
In this section, suppose that 

(3) G := 0{p, g), H := 0(pi, q,) xO{p-p,,q~ q,). 

Here G is the isometry group of := MP^"^ with respect to a nonsingular symmetric 
bilinear form (, ), and H ^ G is the stabilizer of two subspaces U and W of V, where 

(4) V = U(BW, U±W, H\u^O{puq,), H\w ^ 0{p - p^q - qi). 

We parametrize the if-orbits in the isotropic Grassmannian Gicir) of r-dimensional 
isotropic subspaces of V. 

Given any subspace S <^ V, let PuS (resp. PwS) be the projection of S onto the 
[/-component (resp. the VT-component) with respect to V = U Q)W. So 

(5) PuS := (S + W)r]U, PwS := {S + U)r]W. 
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The radical of 5* with respect to the bihnear form ( , ) is 
(6) Rad(5) := 5n5^. 

5 is said to be positive definite if (v, v) > (resp. negative definite if (v, v) < 0) for 
every nonzero vector v G S*. 

Recall that the G-orbit of S can be parametrized by a 3-tuple (s, s+, s_), where 



(7a) 

(7b) Sh 
(7c) s. 



dim Rad(S'), 

dimension of a maximal positive definite subspace of S, 
dimension of a maximal negative definite subspace of S. 



Moreover s + s+ + s„ = dimS". We say that S is of isometry type (s, 5+, s_). 
Now suppose that S is an r-dimensional isotropic subspace of V. Let 

No := {0,1,2, 3,---}. 

The following theorem parametrizes the if-orbit of 5*. 

Theorem 2.1. Let S be an isotropic subspace ofV = MP'^'^ = U (B W. Define 



(8aj 
(8b) 

(8c) 

(8d) 



ru 
rw 

a 
au 



dim(5n f/), 
dim{SnW), 
dim{Rad{P u S)) -Tu 
dim(Rad(PH/S')) — rw, 

dimension of a maximal positive definite subspace of Pc/S* 
dimension of a maximal negative definite subspace of PwS, 
dimension of a maximal negative definite subspace of PuS 
dimension of a maximal positive definite subspace of PwS. 



Then the 5-tuple {ru, rw, a, au, aw) £ Nq^ is H -invariant, and it uniquely determines 
the H -orbit of S. The 5-tuple (r^/, riy, a, a[/, aiy) satisfies the following conditions: 



(9a) 
(9b) 
(9c) 
(9d) 
(9e) 



{ru,rw,a,au,aw) G No^ 

ru + a + au < Pi, 

ru + a + aw < Qi, 

rw + a + aw < P - Pi, 

rw + a + au < Q - Qi- 



Theorem 12.11 shows that {ru, rw, a, au, aw) is an integer point in a convex set of 
^. We denote the if-orbit of S by OG,H{ru, fw, a, au, aw)- 
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Remark 2.2. Obviously, 

(10a) r = dim S = ru + rw + a + au + aw, 

(10b) dimP[/5' = ru + a + au + aw, 

(10c) dimPwS = rw + a + au + aw- 

Proof of Theorem \2.1\ First we prove that {ru^rw^a^au^cLw) is well-defined by 
The subspace S can be expressed as 

k 

(11) 5 = (5nf/)©(5nw^)©0M(ui + Wi), 

i=l 

where Uj G ?7 and Wj G for i = 1, 2, ■ ■ ■ , fc, and 

/c := dim S — dim(S' f\U) — dim(S' fl W) = a + au + aw- 

Then 

k k 

Pt/5 = (^nf/)©0Mu„ VwS = {Snw)®Q^v^i. 

i=l i=l 

Since S is isotropic and U _L W , we have 

5nf/ C Rad(Pc75), 5n C Rad(PvF5). 
Select the vectors Uj + Wj appropriately so that the Gram matrix of {ui, ■ ■ ■ , u^} is 

(12) [ (Ui, U,) ] = 0,xa © hu ® i-hj. 

Then (uj + Wj, + w^) = (uj, u^) + (wj, Wj) = implies that the Gram matrix 

(13) [ (W„ W,) ]f^^^ = -[ (U,, U,) ] = Oaxa © i-Iau ) © 4^ 

In particular, the dimension of a maximal positive definite subspace (resp. maxi- 
mal negative definite subspace) of PuS equals the dimension of a maximal negative 
definite subspace (resp. maximal positive definite subspace) of PwS, and 

dim(Rad(Pt/5)) - dim(5 nU) = dim(Rad(PvK5)) - dim(5 n W). 

So (r^/, rw, a, au, aw) is well-defined by (|H]). 

Second, we show that {ru,rw,a,aij,aw) defined by ([H]) meets conditions ([H]). It 
is obviously if- invariant and in Nq^. By (|T2l) . the subspace Si of U: 

a+a-u 

:= (5nf/)© RUi 

i=l 

satisfies that (v,v) > for every nonzero \ E Si. Let Ui be a maximal negative 
definite subspace of U. Then dim Ui = qi, Si H Ui = {0} and Si + Ui C U . So 

dim(S'i + Ui) = dim Si + dim [/i = ru + a + au + qi < dim U = pi + qi. 



6 HONGYU HE AND HUAJUN HUANG 

Thus (l9bl) holds. Similarly, (Ec]), dMl), and O hold. 

Next we show that every isotropic subspace S' corresponding to (r^/, r^, a, au, aw) is 
in the i7-orbit of S. Parallel to (fTTl) . f[T^ . f[T^ . S' has a decomposition: 

k 

s' = {s'nu)®iS'nw)®^R{u[ + w'^, 

1=1 

where 

Let : Pf/S* — > Pf/S" be a linear bijection such that 

(j){S nu) = s'nu, 0(ui) = for i = i, 2, ■ ■ ■ , fc. 

Then is an isometry. By Witt's Theorem [17], can be extended to an isometry 
gi G 0([/), the orthogonal group of U. Likewise, there is an isometry g2 G 0(W) 
such that 

g^iS nW) = S' nW, g^{wi)=wl for i = l,2,---,fc. 
Then gi x g2 ^ H and it sends 5* to S'. Hence S and S' are in the same if-orbit. 

Finally, given any 5-tuple {ru,rw,ci,ciu,0'w) that satisfies conditions we show 
that there exists an if-orbit corresponding to (r^, r^, a, au, aw)- In this situation S 
is not given. Let {u^, u+ u~J and {wf, ■ ■ ■ , w+_pi, , ■ ■ ■ , w^.^J be 
fixed orthogonal bases of U and W respectively such that for all applicable indices, 

(U^U+) = 1, (uT,uT) = -l, (W+,W+) = 1, (w7,w7) = -l. 

Let S be the subspace spanned by the basis 

(14) {u+ + ur}2i U {w;+w7}^^- u {u+^., + w7^+J^-, U {u;^+, + w+^^,}^- 

Then S* is a canonical isotropic subspace of V and the if-orbit of S is parametrized 
by {ru, rwyCLyO.u, a-w)- We complete the proof. □ 

3. Real Orthogonal Case: Homogeneous Structures of Orbits 

We continue the discussion on {G,H) = (0(p, g), 0{pi,qi) x 0(p — pi,q — qi)). 
First we parametrize the stabilizer Hs of any isotropic subspace SofV. Then we de- 
termine the open if-orbits in GrG'(r). From ([9]) there are only finitely many if-orbits 
in GrG'(T'), and there exists at least one open iY-orbit in GrG(r). 
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Let us parametrize Hs for S G OcHiruTrw^a^au^aw)- Let Pu{rjj,ru + a) be the 
parabolic subgroup of H\u = 0{pi,qi) preserving the partial flag 

UnS C Rad(Pt/S) C [Rad(Pt/^)]^ C (f/ n S)^ (all are restricted in U). 

Let Pw{rw, rw + Q-) be the parabolic subgroup of H\w preserving the partial flag: 

WnS C RadCPwS) C [Rad(PvK^)]^ C (1^ n S)^ (all are restricted in W). 

Then Hs C Pu{rjj,ru + a) x Py^r{rw,rY/ + a). The Levi factor of Pu{ru,ru + a) is 
GL(r[/)GL(a)0(pi — r^/ — a, gi — r^/ — a) and the Levi factor of Pw{rw,r\Y + a) is 
GL{rw)GL{a)0{p — pi — rw — a, q — Qi ~ rw — a) . 

The group must preserve PuS and (Pj/S')-'- (restricted in U). So it preserves 

Rad{PuS) Rad{PuS) 

which are of isometry types (defined by (jTj)) (0, au, clw) and (0, pi — rjj — a — au , qi — 
ru — a — aw) respectively with respect to the induced bilinear form. Moreover, there 
is an orthogonal direct sum 

PuS ^ {PuS)^ [R^d{PuS)]^ 



(15) 



Rad(Pt/^) Rad(Pt/S) 



Rad(Pt/^) 



Choose a basis Bu '■= {ui, ■ ■ • , Up^+g^} of U such that each subspace in 

f/ n 5 C Rad(P{75) C PuS C [Rad(Pt/5)]^ C (f/ n S)^ 

is spanned by the first few vectors in Bu- Then with respect to Bu-, the elements in 
Pui^u, ru + a) are of the form 



(16) 



[An 


Al2 


^13 


Au 


Ai5 


^16" 





A22 


A23 


A24 


A25 


A26 










* 


A35 


A36 








* 




A45 


^46 














A55 


^56 




















where Aii,Aqq G GL{ru) uniquely determine each other, ^22,^55 G GL(a) uniquely 

determine each other, and . E 0{pi—ru — a,qi—ru — a). If (/ii, /12) G Hs 

^ /144 

with hi G Pu{ru,ru + a) and /i2 G -Piy('^VK, ''^vk + a), then /ii must preserve f|T5|) . So 
* in f|T6|) must all vanish, and ^33 x A44 is in the subgroup 0{au, aw) x 0(pi — rj/ — 
a — au, qi — ru — a — aw) of the 0(pi — ru — a,qi — ru — a) factor. 

Similarly, choose a basis Bw '■= {wi, ■ ■ ■ ,^p-pi+q-qi} of such that each subspace 
of W in 

WnS C RadiPwS) C PvK^ C [Rad(PvK^)]^ C (W n S)^ 
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is spanned by the first few vectors in Bw, and furtliermore Uru+i + 'v^rw+i ^ ^ 
i = 1,2, ... ,a + au + aw Tlien witli respect to tlie basis Bw, tlie elements in 
Pwi^W: '^w + a) are of tlie form 



(17) 



[fill 


Bl2 


Biz 


Bu 


-B15 







B22 


B23 


B2A 




B26 








B33 


* 




B3& 








* 


-B44 


-B45 


B46 














-B55 


-B56 

















Bm. 



where Bii,Bqq E GL{rw) uniquely determine each other, B22,B^^ G GL(a) uniquely 

B33 * r\ / \ 

G 0[p - pi - rw - a,q - qi 



determine each other, and 



rw 



a). 



* -B44 

Suppose that {hi,h2) G Hs- Then * in f lTTl) must all vanish, and B33 x is 
in the subgroup 0(aiy, a;/) x 0{jp — pi — rw — a — aw, <i — li — fw — a — au) of the 



0{p — pi—rw — a,q — qi—rw — a) factor. Moreover 

{hi, h2) must preserve the canonical bijection between 
S. 



A22 A23 
A33 



uns 



and 



B22 B23 

533 



smce 



wns 



induced by 



Theorem 3.1. Suppose that {G,H) = (0(p, g), 0(pi,gi) x 0{p — pi,q — qij). Let 
S E OG,Hiru,rw,a,au,aw)- Let hi E Puiru,ru + a) and h2 E Pw{rw,rw + a). Then 
{hi, h2) E Hs if and only if 

(1) the 0{pi — ru — a, qi—ru — a) factor of hi in the Levi decomposition of hi is 
contained in a certain 0{au, aw) x 0{pi — ru — a — au , qi — ru — a — aw)i 

(2) the 0{p — pi — rw — a, q — qi — rw — a) factor of h2 in the Levi decomposition 
of h2 is contained in a certain 0{aw, au) x 0(p — pi — rw — a — aw, Q — Qi — 
rw-a-au); 

A22 A23 
A33 



of hi in ([T6|l and 



(3) the submatrices 
identical. 

In particular, dim Hs can be computed by (fT8|) below, and 



B22 B23 

^33 



of /i2 in ([T71) are 



dimOG,Hiru,rw,a,au,aw) = dim if — dim Hs- 



Proof. When (/ii,/t2) £ Hs, we have verified Theorem 13.11 (1)(2)(3). Conversely, if 
Theorem 13.11 (1)(2)(3) hold, then it is easy to see that {hi, h2) E Hs. 
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It remains to compute dim Hs- By the Levi decompositions, we have 

dim Pi/{ru, ru + a) = -[dimO(pi, gi) + dimGL(r(/) + dimGL(a) 

+ dimO(pi - ru - a,qi - ru - a)], 
dim Pw {r w, fw + a) = ^[dimO(p — pi, g — gi) + dimGL(rvK) + dimGL(a) 

+ dim 0(p — pi — rw — a, q — qi — rw — a)]- 
By Theorem 13.11 (1)(2)(3) and r = ru + ry/ + a + au + aw, we have 
dim. Hs = dim. Pu {ru + a) + dim Pw{rw + o) 



The following corollary is a direct consequence of (1181) . 

Corollary 3.2. // both Ocuiju, rw, a, clu, clw) o,nd Og,h{ju, tvk, o, a'j/, a'y^) exist and 
au + aw = a'u + a'^^, then they have the same dimension. 

Now we describe the open iY-orbits in GrG(r). An if-orbit is open, if and only if it 
has the maximum dimension among all iJ-orbits in GrG'(r), if and only if dimHs is 
minimal for any subspace S in the G-orbit. The next theorem shows that to find the 
open if-orbit(s) OcuiTu ,rw , a, au , aw) in GrG(r), we should maximize au and aw 
subject to the constraints au+aw < r, au < min{pi, q—qi} and aw < niin{gi,p— pi}. 
In particular, a = for all open orbits. 

Theorem 3.3. The open H-orbits in Gicir) are determined as follow: 

(1) When r < minjpi, q — qi} + min{gi,p — pi}, there are several open H-orbits 
and they are parametrized by 

Og,h{0, 0, 0, au, aw) 
with au + aw = r, < au < min{pi, q — qi} and < aw < min{gi,p — pi}; 



— dim 0(pi — ru — a, qi — ru — a) — dim 0{p — pi — rw — a, q — qi — vw — a) 
+ dimO(a;7, aw) + dim 0{pi — ru — a — au, qi — ru — a — aw) 

+ dimO{aw, au) + dimO(p — pi — rw — a — aw, q — qi — rw — a — au) 

— dimGL(a) — dim 0{au,aw) — a{au + aw) 



(18) 




) 



□ 
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(2) When min{pi,g — qi} + mm{qi,p — pi} < r < mm{p,q}, there is a unique 
open H- orbit and it is parametrized by 

Og,h{j — ajj — aw, 0, 0, ajy, aw) if dimt/ > dimVI^, 
CG,iT(0, r — ajj — aw, 0, a[/, aw) if dim f/ < dim , 

with ajj = min{pi, q — qi} and aw = min{gi,p — Pi}- 

Proof. Let Og,h{ju, rw, a, au, ^w) be an open orbit and S G Ocni^u, ^w, clu, aw)- 
Then the stabihzer Hs must be of minimal dimension among all Hs' for S' G Gicir). 

First, we prove that a = 0. If on the contrary a > 0, then (r^/, r^y, 0, ajy + a, aw) 
meets conditions and there exists S' G Ocnii^Uyf^Wy^yau + a, aw)- By (fT5]l . we 
compute in MAPLE that 

dim Hs — dim Hs' = + 1) > 0. 

Therefore dimif^ > dim if 5/. It contradicts the assumption that S is in an open 
orbit. Thus a = 0. 

Second, we show that r^/ = or rw = 0. If this is not true, then ru > 0, rw > 0, 
a = 0, and r = ru + rw + au + aw- The 5-tuple (ru — l,rw — 1, 0, au + l,aw + 1) 
meets conditions (jH]) and thus there exists 5" G Oq ni^u — l,rw — ^,0,au + l, aw + ^)- 
By dH, 

dim Hs — dim Hs' = p + q — r — au — aw — l>p + q — 2r>0. 

It contradicts the assumption that Hs is of minimal dimension. Therefore ru = or 
rw = 0. 

Third, suppose that r < min{pi, q — qi} + min{gi,p — pi}. We claim ru = rw = 0. 
If not, without loss of generality, assume ru > 0. Obviously, rw = and a = 0. By 

au + aw < r = ru + au + aw < min{pi, q - qi} + min{gi,p - pi} 

we get au < min{pi,g — qi} or aw < min{gi,p — pi}. If au < min{pi,g — qi}, 
then {ru — 1, 0, 0, au + 1, aw) meets conditions and there exists S' G OcHiTu ~- 
l,0,0,a^ + l,aH/). By (USD, 

dim Hs - dim Hs' = (p - Pi + q - qi - au - aw) + (r^/ - 1) > 0. 

If aw < iiiin{gi, p— pi}, the same argument works for S' G Ocni^u — 1, 0, 0, au, aw + 
1). So Hs is not of the minimal dimension for S G Grdr). We reach a contradiction. 

Therefore, when r < min{pi,g — qi} + min{qi,p — pi}, we have ru = rw = a = 
and au + aw = r. Moreover au < min{pi,g — qi} and aw < min{gi,p — pi} by 
([9]). This proves the necessary part of Theorem 13.31 (1). Conversely, all orbits with 
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fu = fw = a = and au + aw = f have the same dimension by Corollary 13.21 So 
they are open orbits. This proves the sufficient part of Theorem 13.31 (1). 

Finally, suppose min{p, q} >r > min{pi, q — qi} + min{gi,p — Pi}. 

(A) Claim: r^/ = when dimf/ < dimly, and r^i = when dimly < dimU. 

Suppose that dim?7 < dimpy. If on the contrary ru > 0, then rw = 
and a = by the preceding arguments. From fl9bl) and fl9cl) . 

2ru + au + aw < Pi + qi = dimU < dimpy = (g - qi) + (p - Pi)- 

So au < q — qi or aw < p—pi- Without loss of generality, suppose au < q — qi- 
Then {ru — 1, 0, 0, au + 1, aw) meets conditions ([9]) and thus there exists 
5" G Ocniru - 1, 0, 0, au + 1, aw). By ([H]), we have 

dim Hs - dim Hs' = (p - Pi + q - qi - au - aw) + (^t/ - 1) > 0. 

This contradicts the assumption that Hs is of minimal dimension. Therefore, 
ru = when dim U < dim W. Similarly, rw = when dim W < dim U. 

(B) Claim: au = min{pi,g — qi} and aw = min{gi,p — pi}- 

Without loss of generality, we assume that dim U < dim ly . Then ru = 
and a = 0. Hence r = rw + au + aw > minjpi, q — qi} + mm{qi,p — pi}. Con- 
ditions ([9]) show that au < min{pi, g — gi} and aw < iiiin{gi,p — pi}. Suppose 
on the contrary au < mm{pi,q — qi} (similar for aw < niin{gi,p — pi}), then 
rw > and the 5-tuple (0, rw — 1, 0, au + 1, aw) meets conditions ([9]). There 
exists S' e Og,h{0, tw - 1, 0, au + 1, aw)- By (fTHl), 

dimi^s — dim Hs' = (pi + qi — au — aw) + {rw — 1) > 0. 

This contradicts the assumption that Hs is of minimal dimension. Therefore, 
au = min{pi,g — gi} and aw = iiiin{gi,p — pi}. Moreover, rw = r — au — aw- 

The preceding argument proves the necessary part of Theorem 13.31 (2). Conversely, 
Theorem 13.31 (2) uniquely determines one if-orbit, which must be the open if-orbit 
of Gicir)- This proves the sufficient part. □ 

Example 3.4. Suppose that 

<Pi,qi <r <p-pi,q- qi. 

We describe the open H -orbits in GrGr(r) by Theorem \3.3[ 

(1) When pi, gi < t < pi + gi(= dim U), the open orbits in GrG<(r) are 

OG,H{0,0,0,au,r - au) for r - qi < au < Pi- 
Totally, there are (pi + gi + 1 — r) many open orbits in Gicir). 
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(2) When pi + qi < r < p — pi,q — qi, the open orbit in GTcir) is unique and it is 

OG,HiO,r-pi - gi,0,pi,gi). 

4. Real Orthogonal Case: Decompose an if-ORBiT into (H n Go)-oRbits 

Throughout this section, let pi, qi,p — pi,q — qi > and < r < mm{p, q}. The 
orthogonal group G = 0{p, q) and its subgroup H = 0{pi, qi) x 0{p — pi, q — qi) have 
4 and 16 connected components respectively. Let Gq denote the identity component 
of G. We are interesting in the decomposition of an if-orbit in GrG'(r) into (HnGo)- 
orbits. This gives the parametrization of 

{H n Go)\GiGir) ~ (/f n Go)\Go/P 

where P is a maximal parabolic subgroup of Gq. 

It turns out that every if-orbit in Grdr) may decompose into 1, 2, or 4 (if fl Gq)- 
orbits. Moreover, an open ii-orbit in Gicir) is also an open {H fl Go)-OThit except 
for the case p = q = r, in which the unique open if-orbit in Gicir) decomposes into 
2 open (if fl Go)-orbits. 

For the pair [G, H) acting on V = U ® W, we fix certain orthogonal bases 
Wi^ ■ ■ ■ , u+ , uj", ■ ■ ■ , u~ } of t/ and {w+, ■ ■ ■ , w+_p^, w^", ■ ■ ■ , w'.^J of W respec- 
tively, such that for all applicable indices, 

^t) = 1. (uj, uj) = -1, (w+, w+) = 1, (w7, W-) = -1. 

These vectors in the same order form a basis By of V. Denote the matrices 

(19) lt^-=In and i(~) :=i„_i©(-ii). 

Then with respect to the basis By, the 4 connected components of G are represented 
by elements Ip^^ © Ip^^ © ® ^q-\i ^i)^2 € {+, — } • and the 16 connected 
components of H can be denoted by H^^Jl where ti,t2,^3,i4 £ {+, — } and 

Moreover, if fl Go consists of 4 components: 

(20) if n Go = ff ++ U ff +1 U ff 1+ U HZZ , 
and ff/ (ff n Go) consists of 4 cosets: 

(21) H/{H n Go) = {HttiH n Go), H+-{H n Go), H-+{H n Go), H--{H n Go)} . 

Let S spanned by (fT4l) be the canonical subspace of an ff -or bit Ocuiju^ ^w, ci,CLu, ciw) 
in GTcir). The (ff nGo)-orbit and the ff -orbit of S are related by the canonical maps: 

(22) (ffnGo)-S ~ (ff nGo)/(ff nGo)5 

= (ff nGo)/[ffsn(ff nGo)] 

~ [HsiHnGo)]/Hs 
C H/Hs ^ H-S. 
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So the number of {H fl Go)-orbits in the if-orbit of S is equal to [H : Hs{H fl Go)], 
which by fl^Tl) is equal to 4/m, and m is the number of cosets of H/ [HnGo) — namely 
H^:l{HnGo), Hl-{HnGo), H-^{HnGo), and //""(i/ n Gq) — that intersect Hg. 
In particular, the if-orbit of S decomposes into only 1, 2, or 4 (iJ fl Go)-orbits. 

The 5-tuple {ru,rw,ci,ciu,ciw) satisfies conditions Q). The number of cosets of 
H/ {H n Go) that intersect S can be determined by {ru, rw, a, clu, clw) as follow: 

(1) Claim: If ru + a + au < Pi or rw + a + aw < P — Pi, then Hs intersects 
H~t{HnGo). 

If rw + a + aw < P — Pi in fl9d|) . then Wp„p^ does not appear in the basis 
f|T4l) of S (i.e. Wp_p^ G S*-*-). Let L G GL(V) have -1 eigenvector Wp_p^ and 
+1 eigenvectors in i3y\{w+_pj. Then Le Hgn H^X ^HsH [H-^iH n Go)] 
since H++ = H^+Hzt- 

Similar for ru + a + au < pi in (l9b|l . 

(2) Claim: li ru + a + aw < Qi or rw + a + au < q — Qi, then Hs intersects 
Ht^iHnGo). 

The argument is similar to the preceding one. 

(3) Claim: If rj/ > or rvi/ > or a^/ > or aw > 0, then Hs intersects 
H--{HnGo). 

If au > 0, then S has a basis vector + w^- in ffl^ for some Let L G 
GL(y) have -1 eigenvectors in {u^, wj} and +1 eigenvectors in i3y\{u^, w~}. 
Then LeHsH H^+ CHsH [H^+{H n Gq)] since = H^^H+I. 

Similar for the other cases. 

Obviously, Hs always intersects H^+iH n Gq) = H n Gq. These imply the following 
theorem. 

Theorem 4.1. The number N of {H fl Go)-orbits in an H-orbit is given below: 



N 


G 


H 


H-orbit 


conditions 


4 


0{2a, 2a) 


0(a, a) X 0(a, a) 


OG,H(0,0,a,0,0) 




2 


0(p,2a) 


0(pi,a) X 0(p-pi,a) 


CG,H(0,0,a,0,0) 


pi > a, p — pi > a, p > 2a 


0{2a, g) 


0(a, 5i) X 0(a, g - qi) 


Cg,h(0, 0,a,0, 0) 


q\ > a, q — q\ > a, q > 2a 


0(p,q) 

(p = g) 


0(pi,(ji) X 0(p - - qi) 




ru -\-a-\-au = Pi, 

rw + £1 + aw = P — Pl, 

+ a + aw = <?i, 
rw + a + ajj = (J - gi, 
^[7 + rw + 1(7 + aw > 


1 


all the other situations. 



Theorem 14.11 together with Theorem 13.31 provides the following decomposition of 
an open if-orbit into open {H fl Go)-orbits. 

Corollary 4.2. An open H-orbit in GrG!(r) always decomposes into 1 open (HnGo)- 
orbit except for the case p = q = r, in which the unique open H-orbit decomposes into 
2 open {H fl Gq) -orbits. 

Proof. Let OcuiTu, rw, a, au, ^w) be an open if-orbit in GrG(r). By Theorem l3.3l we 
have a = and au + aw > 0. So by Theorem 14.11 Ocuiju, rw, a, au, aw) decomposes 
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into 1 or 2 open {H fl G'o)-orbits, and it decomposes into 2 {H n G'o)-orbits only when 
ru + au=Pi, rw + aw=p-Pi, ru + aw = Qi, rw + au = q - qi- 

This together with r = ru + rw + au + aw imphes that p = q = r. In such case, 
Theorem 13.31 shows that the open if-orbit is unique. □ 

Example 4.3. We describe the open H-orbits that each decomposes into 2 open 
{H n Go)-orbits for dim?7 < dimW^. In such case, G = 0{r,r), H = 0{pi,qi) x 
0(r — pi, r — qi), and Pi + qi < r. The open H-orbit is Og,h{0, r — pi — qi, 0,pi, qi) 
in Gicir). The canonical subspace S of the H-orbit is spanned by f[T^ .- 

f + I -i''-pi-gi iif+, - iPiiif-i + T?i 

We change the sign of one of the vectors and wf in the above basis vectors, and 
let S' be the subspace spanned by the new set of vectors. Then S and S' are in the 
same open H-orbit but in the two different open {H fl Go)-orbits. 

5. Unitary Case 

In this section, let 

G:=\J{p,q) and if := U(pi, gi) x U(p - pi, g - gi), 

where G is the isometry group of 

y = f/ © ~ C^i+^i © CP-pi+1-11 (orthogonal direct sum), 

and H the subgroup of G that stabilizes U and W. We explore the if-action on the 
Grassmannian Grcir) of r-dimensional isotropic subspaces of for r < min{p, g}. 
Most results are completely parallel to those in real orthogonal cases (see Sections 
2, 3, and 4). We sketch some proofs and omit other proofs that are routine. A 
distinction between unitary and real orthogonal cases is that the unitary groups here 
are connected and reductive, while their real orthogonal counterparts are disconnected 
and semisimple. However, the difference makes no impact in our analysis. 

Theorem 5.1. The H-orbit of an isotropic subspace S G Gicir) can be completely 
parametrized by the H -invariant 5-tuple (r^y, rvF, a, a[/, ovk) defined as in ([8]). The 
5-tuple (rj/, rvK, a, 0(7, oiy) satisfies the same constraints as in Q. 

Denote the if-orbit of 5* by Og,h{j'u, rw, a, au, aw)- 

The stabilizer Hs has similar structure as in Section 3. An element of Hs can 
be written as (/ii,/i2) with hi G Pu{ru,ru + a) and /i2 G Pw{rw,rw + a), where 
Pu{ru,ru + a) is the parabolic subgroup of H\u preserving the partial flag 

UnS C Rad(Pt/5) C [Rad(Pt/^)]^ C (f/ n S)^ (all are restricted in U), 
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and Pw{rw,rw + a) is the parabolic subgroup of H\w preserving the partial flag 
WnS C Rad(PvK5) C [Rad(PvK5)]^ C {W n S)^ (all are restricted in W). 
Moreover, hi preserves PuS, /12 preserves PwS, and (/ii,/i2) G Hs preserves the 
canonical bijection (induced by S) between 

Let Bu 



PuS 



and 



PwS 



uns wns 

{ui, • ■ ■ , Up^_|_q^} be a basis of U such that the subspaces in 
UnSC Rad{PuS) C PuS C [Rad(Pt7^)]^ C (f/ n S)^ 

Let Bw ■= {wi, ■ ■ ■ , w. 



are spanned by the first few vectors in Bu 
basis of W such that the subspaces in 



p-pi+q 



-gj be a 





Au 


^13 


Al4 


Ai5 


AiQ- 






B12 


-B13 


-B14 


Bi5 


Big' 





A22 


A23 


A24 


A25 


A26 







B22 


-B23 


-B24 


B25 


B2& 








A33 





A35 


A36 


, h2 = 








-B33 





B35 


B36 













A45 


Aie 











-B44 


-B45 


Bi6 














A55 


A56 
















B55 


B^e 

















Am. 



















Bee. 



wns C Rad{PwS) C PwS C [Rad(PvK^)]^ ^ {W n S)^ 

are spanned by the first few vectors in Bw, and furthermore Ur^+i + ^rw+i ^ S for 
i = 1,2, ■■■ ,a + au + aw Then hi and /i2 have the following matrix representations 
with respect to Bu and Bw respectively: 
(23) 



hi 



Here An, Aqq G GLj.^(C) uniquely determine each other; ^22, ^55 G GLa(C) uniquely 
determine each other; A33 x ^44 is in a subgroup \J{au, aw) x U(pi —ru — a — au, qi — 
Tu — a — aw) of the U(pi — ru — a,qi — ru — a) factor in the Levi decomposition 
of Pu{ru,ru + a); Bii,Bqq G GLr^y(C) uniquely determine each other; 522,-855 £ 
GL„(C) uniquely determine each other; x i?44 is in a subgroup \J{aw, du) x U(p — 
Pi—rw — a — aw,q — qi—rw — o. — au) of the U{p — pi — rw — a, q — Qi — rw — 0) factor 

in the Levi decomposition of Pwijw, rw + 0). Moreover 

Theorem 5.2. Let S G OG,H{Tu,rw,o,,au,aw)- Then {hi,h2) G Hs for hi G 
Pui^u, ru + a) 0?^^^ ^2 G PvK(^vy, rw + a) if and only if 

(1) the \J{pi—ru —a, qi—ru — a) factor in the Levi decomposition of hi is contained 
in a certain \J{au, aw) x U(pi — ru — a — au, qi — ru — a — aw); 

(2) the U{p — pi — rw — 0',q — qi ~rw — o) factor in the Levi decomposition 0//12 is 
contained in a certain ^{aw, au) x \J{p—pi—rw — a — aw, q — qi — f'w ~(^~0'u) i 



' A22 


^23" 




-B22 


B23 





^33. 







B33 



(3) the suhmatrices 



A22 




A23 
A33 



of hi and 



B22 




-B23 
B33 



of h2 in fl23|) are equal. 
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In particular, dimCG,i?('"!7, tw, a, du, dw) — dimH — dim Hs and 

dim Hs = (pi + qif + {p-pi + q- qif - 2(pi + qi)ru -2{p-pi + q- qi)rw 
5 5 

(24) +7;^u + o^w + ia + au + aw)i-'2p - 2q + Ar - au - aw)- 



The above dimension of Hs is computed similarly as in real orthogonal case. 

Corollary 5.3. // both Og,h{j'u, f'Wi au, aw) and Og,h{tui fWi a, a'^, a'w) exist and 
ajj + aw — a'[7 + ^^^'^ ^^^V have the same dimension. 

Theorem 5.4. The open H-orbits in Groir) are: 

(1) When r < min{pi, q — qi} + min{g'i,p — pi\, there are several open H-orbits 
and they are parametrized by 

Og,h{^, 0, 0, au, aw) 

with au + aw — r, < au < min{pi, q — qi} and < aw < min{gi,p — pi}; 

(2) When min{pi,g — qi} + min{gi,p — pi} < r < min{p,q}, there is a unique 
open H-orbit and it is parametrized by 

OcuiT — au — awi 0, 0, au, aw) if dimU > dimW , 
Og,h{0, r — au — aw, 0, au, aw) if dimU <dimH^, 

with au — min{pi, q — qi} and aw — niin{gi,p — Pi}- 

6. Orthogonal Case over k 

Let G :— 0„(C) andi? := 0^(C) x 0„_^(C), where G is the isometry group of 

V ®W ^C"" ® C"-"' (orthogonal direct sum), 

and H the subgroup stabilizing U and W. We study the if-action on Gioir) (r < 
n/2), the Grassmannian of r-dimensional isotropic subspaces of V. All results in this 
section may be extended to the orthogonal pairs (G, H) = (0„(k), Om(k) x 0„_^(k)) 
over an algebraic closed field k of characteristic not 2. 

Theorem 6.1. The H-orbit of an isotropic subspace S G GrG(r) can be completely 
parametrized by the H -invariant 4-tuple {ru,rw,a,b) defined by 



(25a) 


ru 


:= dim(^n?7), 


(25b) 


rw 


:= dim(^nW^), 


(25c) 


a 


:= dim(Rad(Pt/5)) -rc7 






= dim(Rad(P^yS')) — rw, 


(25d) 


b 


:= dim PuS - dim(Rad(Pt75)) 






= dimPvi/>S' — dim(Rad(PvK>S')) 
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The 4-tuple {ru, rw, a, b) satisfies rjj + rw + a + b = r and the constraints: 

(26a) {ru,rw,a,b) e No^ 

(26b) 2ru + 2a + b < m, 

(26c) 2rw + 2a + b < n - m. 

We denote the H -orbit of S by OG,H{ru,rw,a,,b). 

Proof. The well-definedness of {ru,rw,Q-,b) by (|25|) is done similarly as in Theorem 
12.11 Obviously (r^/, r^y, a, 6) is if-invariant. We have S* fl ?7 C Rad(Pt/5') C PjjS C 
[Rad(Pt75)]^ in U. So 

m-ru-a = dimf/ - dimRad(Pt/^) = dim{[Rad(Pc/^)]^ H f/} 
> dimPijS = ru + a + b. 

So fl26bl) holds. Likewise (EEcl) holds. 

Let 5" be another isotropic subspace corresponding to the same 4-tuple {ru, rw, a, b). 
There exist orthonormal bases for Radjp'^g) ^"^^ Rad(Py5') since C is algebraic closed 
with characteristic not 2. So S" and S' have decompositions 

t t 

5 = (5 n f/) © C(ui + Wi), s' = (s' n f/) © C(u^ + wO, 

i=l 1=1 

where the Gram matrices 

[(W-W.)],x,= [(W^,W;.)],,, = Oa.a(B{-h). 

Define an isometry (p : S ^ S' that sends SnU to S' HU and Uj to u- for 2 = 1, ■ ■ ■ , t. 
Then extend to an isometry gi of U by Witt's Theorem [17]. Similarly, there is 
an isometry g2 of W that sends S H W to S' H W and Wj to for i = 1, - ■ ■ ,t. 
Then gi x g2 E H sends 5" to S'. This shows that the 4-tuple {ru, rw, a, b) uniquely 
determines one if-orbit. 

Finally, given a 4-tuple {ru,rw,CL,b) satisfying conditions ( l26ll . we shall prove that 
there exists an if-orbit corresponding to {ru,rw,a,b). Let Bu '■= {ui,--- , u^} be 
an orthonormal basis of U , and Bw '■= {wi, ■ ■ ■ , w„_m} an orthonormal basis of W. 
Let S be the subspace spanned by the basis 

(27) {ui + ium+i-i}Zi U {w,- + u {u 

U{Ur^+fc + iUm+l-ru~k + ^rw+k + '^^n-m+l-rw-k}k=l 

in which i := V— 1. Then S" is a canonical isotropic subspace of representing the 
if-orbit that corresponds to the 4-tuple {ru, rvi/, a, &)• □ 
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Now consider the stabilizer Hs- An element of Hs can be written as (/ii, h2) with 
hi e Pu{ru,rij + a) and /i2 G Pwixw^rw + a), where Pu{ru,ru + a) C is the 
parabolic subgroup preserving the (partial) flag 

Ur\S<Z Rad(P[/5) C [Rad(P[/5)]^ C ([/ n S)^ (all are restricted in f/), 

and Pvf(^vi^? ''"w^ + a) C is the parabolic subgroup preserving the (partial) flag 

Wr\S<Z Rad(Pv^5) C [Rad(Pv^5)]^ C (py n S)^ (all are restricted in W). 

Moreover, h-i preserves 'PuS, /i2 preserves 'PwS, and hi x /i2 preserves the canonical 

biiection (induced by S) between ^ ^ and —r—^- 

Let Bu := {ui, ■ ■ ■ , Up^+g^} be a basis of t/ such that each subspace in 

UnSC Rad(PuS) C PuS C [Rad(Pt/5)]^ C {U f] S)^ 

is spanned by the first few vectors of Bu- Let Bw '■= {wi, • • • , Wp-pj+^-qj} be a basis 
of W such that each subspace in 

is spanned by the first few vectors of Bw, and in addition u^^+j + w^^y+j G 5* for 
i = 1,2, - • • ,a + b. Then /ii and /i2 have the following matrix representations with 
respect to Bu and Bw respectively: 
(28) 



hi 



Here An, Aqq e GLr^{C) uniquely determine each other; A22, A55 G GLa(C) uniquely 

of the 



r^ii 


A12 


^13 


Au 


^15 






r^ii 


B12 


-B13 


-B14 


-B15 


Biq' 





A22 


^23 


A2A 


^25 


^26 







B22 


-B23 


-B24 


-B25 


B26 








Az3 





^35 


^36 


, /i2 = 








-B33 





-B35 


BsG 











A44 


A45 


^46 











Bu 


-B45 


B46 














A55 


^56 
















-B55 


B56 

















^66. 



















Bm. 



' A22 


A23 




-B22 


-B23 





A33. 







-B33_ 



determine each other; ^433 x ^444 is in a subgroup ©^(C) x Om-2ru-2a- 
Om-2ri7-2a(C) factor in the Levi decomposition of Puij-u, ru + a); Bu, Bqq e GLr^^(C) 
uniquely determine each other; 522,-855 € GLa(C) uniquely determine each other; 
-B33 X i?44 is in a subgroup 06(C) x On~m-2rw-2a-b{'C) of the On-m-2nv-2a(C) factor 

in the Levi decomposition of Pw{rw, "^w + a)- Moreover 

Theorem 6.2. Lei S G Ocni^u-, "^w, o,, b). Then {hi, /12) G Hs if and only if 

(1) hi e Pu{ru,ru + a) and /12 e Pi4/(riy,rvy + a). 

(2) the Om-2ru-2a{^) factoT of hi in the Levi decomposition of hi is contained in 
a certain ©^(C) x 0„,-2ru-2a-b{'^); 

(3) the 0„_,„_2rM/-2a(C) factoT of h2 in the Levi decomposition o//i2 is contained 
in a certain Ob{C) x On-m-2rw-2a-b{'C); 
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(4) the submatrices 



A22 A23 
A33 



of hi and 



B22 B23 

533 



of h2 in ( 128]) are equal. 



In particular, dimOcHi^u^ a, b) = dim/7 — dimif^ and 
(29) dimHs = 7tU +o n +ir + ^+L -«^ 



2V2y 2V2y 2 2 V2 
1 / m — 2ru — 2a\ 1 /n — m — 2rvi/ — 2a 
2V 2 J"2'^ 2 
m — 2r[/ — 2a — 6\ fn — m — 2rw — 2a — b 
2 r\ 2 



+ 



The expression of dimHs is in the same form as in the real orthogonal case ( fTSj) 
except that we use b in place of au + aw here. We can similarly prove the following 
theorem regarding the open iZ-orbits (c.f. Theorem 13.31) . 

Theorem 6.3. Ifr < min{m, n— m}, then the unique open H -orbit is (9g,h(0, 0, 0, r) 
z/min{m, n — m} < r < ^, the unique open H -orbit is 

Ocnir — n + m, 0, 0, n — m) if dimf/ > dim PF, 
C^g,h(0, r — m, 0, m) if dimf/ < dim 14^. 

Next we discuss how an Tf -orbit decomposes into (-ffflGo) -orbits. Let I^^ := and 
I^^^ := In~i © {—h) as in (fTSl) - The group G = 0„(C) has 2 connected components 
Go = SO„(C) and J^^Gq. When < m < n, the subgroup H = 0^(C) x 0„_™(C) 

Ttl 

hi 



has 4 connected components Hll where ti,t2 G {+, — } and 



The subgroup H (1 Gq has 2 components iJ^ and ifZ; the quotient 
H/{HnGo) = {HXiHnGo), H-{HnGo)}. 

Let 5* be the isotropic subspace of OG,H{ru,r\Y,a,b) spanned by fl27j) . By the same 
argument as in (|22|) . the if-orbit of 5* decomposes into 

[H:Hs{HnGo)] e {1,2} 

many (/f nGo)-orbits. Moreover, the if-orbit of S decomposes into 2 (i/nGo)-orbits 
if and only if Hs has no intersection with H^{H fl Gq). This could be determined by 
the basis ([27]) of S. 

Theorem 6.4. Every H-orbit of an isotropic subspace decomposes into 1 or 2 {H fl 
Go)-orbits. Moreover, the H-orbit OG,H{ru,rw,ci,b) decomposes into 2 {H n Gq)- 
orbits if and only if the equalities (]26bp and fl26cp hold, that is, r + a = n/2 where 
r = ru + rw + a + b. 

The proof is similar to the discussion preceding Theorem 14.11 and is omitted. 
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Corollary 6.5. An open H-orbit decomposes into 2 open {H (1 Go) -orbits if and only 
if r = n/2. 

Finally, we show by one example in the orthogonal case that the canonical map / 
defined in ([2]) is neither surjective nor injective. 

Example 6.6. Let the symmetric pair {G, H) = (08(C), 04(C) x 04(C)) act on 
V = U ®W = ^C^. Let {ui, U2, U3, U4} and {wi , W2,W3,W4} be orthonormal 
bases of U and W respectively. Let J^'q denote the partial flag variety of isotropic 
subspaces of dimensions 1 and 2. Define the canonical map 

/: H\T'a - H\Gig{1) x i/\GrG(2). 

(1) (f is not surjective) Let 

Ti = C(ui + iu2) and T2 = C(wi + iw2) + C(w3 + iw4). 

Then H ■ Ti consists of isotropic lines in U and H ■ T2 consists of isotropic 
2-dimensional subspaces in W . So {H ■ Ti,H ■ T2) is not in the image of f 
since there is no flag {Gi C C2} in T'q such that Gi E H ■ Ti and G2 G H ■T2. 
Therefore f is not surjective. 

(2) (f is not injective) Denote the isotropic subspaces 

Si = S[ = C(ui + iu2 + iwi - W2), 

5*2 = Si + C(ui - iu2 + iwi + W2) + C(u3 - iu4 + iw3 + W4), 

^2 = 5*^ + C(U3 + iU4 + iW3 - W4) + C(U3 - iU4 + iW3 + W4). 

Then Si and S[ are in the same H-orbit, and S2 and S'2 are in the same 
H-orbit by Theorem \6.1\ Consider the following two flags in J-'qI 

F = {SiC S2}, F' = {S[ C S'2}. 

We see that f{F) = f{F'). 

If F and F' are in the same H-orbit, then there is h E H such that h- Si = S^ 
for i = 1,2. Then 

h ■ [PuSi n {PuS2)^] = PuS'i n {PuS'2)^. 

However, PuSin{PuS2)^ = {0} butPuS[n{PuS'2)^ = PuS[ = C(ui + iu2). 
We reach a contradiction. Therefore, F and F' are not in the same H-orbit. 
This together with f{F) = f{F') shows that f is not injective. 
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7. Symplectic Case 

Let ^ be a 2n-dimensional real vector space equipped with a symplectic form ( , ). 
Let G or sometimes G{2n) be the symplectic group preserving (, ). Let H be the 
stabilizer of two subspaces U and W of V, where 

(30) V = U®W, U ±W, H\u = G{2m), H\w = G{2n - 2m). 

Let 5* be a r-dimensional isotropic subspace of V. Obviously, r < n. Denote the 
r-dimensional isotropic Grassmannian by Gicir). Consider the H action in GrG'(r). 
By Matsuki's theorem, there are only finite number of if-orbits in Gvcir). We would 
like to give an elementary parametrization of these orbits, compute their dimensions 
and describe the stabilizers. When r = n, H action on the Lagrangian Grassmannian 
Gicin) is described in [2]. 

The structure of 5* in reference to the decomposition U (B W can be described as 
follows: 

k 

5 = (5 n f/) © (5 n w^) © 0M(u, + w,), 

i=l 

with Ui E U and Wj G W. In particular, 

k 

(31) PuS := {s + w)nu ={SnU)®^Rui, 

1=1 

k 

(32) PwS := {s + u)nw = {SnW)®^Rwi. 

1=1 

Notice that these decompositions are NOT cannonical. The vectors Uj and Wj are by 
no means unique. Observe that ^^^i Kuj can be decomposed into two subspaces: a 
radical and a nondegenerate subspace, with respect to ( , ) . 

Theorem 7.1. Let S be an isotropic subspace ofV = U(B W. Let 

ru = dim(S'nf/), 

rw = dim{SnW), 

a = dim(Rad(P(/5')) — ru, 

b = dim(P[/S') — a — r^. 

Then b must be even, a + b + rjj + r^y = dim S = r and 

a = dim(Rad(Pvi/5')) — r^y, b = dim{P]yS) — a — ryy. 

The 4-tuple {ru,r\y,a,b) uniquely determines the H -orbit of S. In addition, an 
isotropic S parametrized by the 4-tuple {ru,rw, a,b) G Nq^ exists if and only if b 
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is even and 

(33a) Tu + a + < m, 

(33b) rw + a + - < n — m. 



Proof. Consider the restriction of the symplectic form on ^^^^^^Muj. In reference to 
( I3T1) . we have 

k 

Rad(Pc/5) = Rad(0 Mu,) ®{Sr\U). 

i=l 

So a := dim(Rad(P(75')) — ru = dim(Rad(0 Muj)). Hence h := dim(P(/S') ~a — ru = 
dim[(0Muj)/Rad(0Muj)] must be even dimensionaL In addition 

r = dim(S') = dim(S' nU) + dim{S nW) + k = ru + rw + a + b. 

Define a linear isomorphism $ from ^^^i to by letting $(uj) = Wj for 

every i. Since 0jL]^ IR(uj + Wj) is isotropic, $ must negate the symplectic form ( , ) 
restricted to the two linear subspaces. Hence 

k k 

$(Rad(0Mui)) = Rad(0Mwi). 

i=l i=l 

So a = dim(Rad(Pi^5')) — r^^ and b = dim(Pvi/5') — a — riy. In addition, a La- 
grangian subspace of (0Muj)/Rad(0]Ruj) induces an isotropic subspace in U by 
adding Rad(P(75'). We have ru + a + ^ < m. Similarly, we have rw + a + ^<n — m. 

Now given [ru, rw, a, b) satisfying the conditions fl33|) . we construct an r-dimensional 
isotropic subspace as follows. Fix standard bases {ei, 62, ... , e^, fi, f2, . . . fm} in U 
and {e,m+i, 6^+25 ■ ■ ■ ,^n, fm+i, fm+25 • • • fn} in W. Let 5* be the canonical isotropic 
subspace spanned by 

(34) {ei}^^_^^^i U {ej}J^^_^^.^^ U {e^ + f„+^, + e^+jj^ U {e^ + em+k}l=t/l+i ■ 
It is clear that S represents the if-orbit parametrized by {ru, rw, a, b). 

It remains to show that (r^/, rw, a, b) parametrizes a single if-orbit. This can be 
done similarly as in the proof of Theorem 12. 1[ □ 

Suppose that b is an even nonnegative integer. Suppose that rj/ + a + | < m 
and rw + a + | < n — m. Let Ocui^u, rw, a, b) or 0{ru, rw, a, b) be the iJ-orbit 
parametrized by (r^/, rw, a, b). Let r = ru+rw+a+b. Then 0{ru, rw, a, b) C GTcij). 
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Now fix G 0{ru,rw,ci,b). We would like to compute its stabilizer Hs- Let 
Pu{fUi fu + o) be the parabolic subgroup oiG{U) preserving the (partial) flag 

UnSQ Rad(P[/,5) C Rad(P[75)^ C ([/ n (aU are restricted in C/), 

and Pw{rw: fw + the parabohc subgroup of G{W) preserving the (partial) flag 

WnSC Rsid(PwS) C Rad(Pw^5)^ C (W^ n S)^ (aU are restricted in W). 

The Levi factor of Pui^u-i^u + «) is GL(rc/)GL(a)G(2m — 2ru — 2a) and the Levi 
factor of Pw{rw: tw + a) is GL{rw)GL{a)G{2n — 2m — 2rw — 2a). 

With the settings, an element of Hs can be written as {hi, for hi e Pu{ru, ru + a) 
and h2 G Pw{j-y^,ry/ + a). Moreover, hi preserves 'PuS, /i2 preserves PvfS", and 

hi X /i2 preserves the canonical bijection (induced by S) between ^ ^ and ^ 



uns 



Wf\S' 



Let := {ui, • • • , Up-^+^^j be a basis of U such that each subspace in 

UnS(Z Rad(Pf/5) C P^5 C [Rad(Pt;5)]^ C ([/ n S)^ 

is spanned by the first few vectors of Bu- Let Bw ■— {wi, • • • , Wp-p^+q-g^} be a basis 
of W such that each subspace in 

WnS C Rad(PvK^) C Pvi.^ C [Rad(Piy5)]^ C {W n S)^ 

is spanned by the first few vectors of Bw, and in addition Ur^+i + w^^^+j G S for 
i = 1, 2, • ■ • ,a + b. Then /ii and /i2 have the following matrix representations with 
respect to Bu and Bw respectively: 
(35) 



hi 



Here All, Aqq G GL^^ (C) uniquely determine each other; ^22, ^55 G GLa(C) uniquely 
determine each other; A33 x A44 is in a subgroup G(&) x G{2m — 2ru — 2a — b) of 
the G{2m — 2ru — 2a) factor in the Levi decomposition of Pu{ru, ru + a); Bu, Bqq G 
GLj.j^(C) uniquely determine each other; B22, -B55 G GLa(C) uniquely determine each 
other; B33 x B44 is in a subgroup G(b) x G{2n — 2m — 2rw — 2a — b) of the G{2n — 
2m — 2rw — 2a) factor in the Levi decomposition of Pwirw^T^w + o)- Moreover 



r^ii 


A12 


^13 


Au 


^15 


Aie' 




[Bii 


B12 


Bi3 


Bu 


Bl5 


Big' 





A22 


A23 


A24 


A25 


A26 







B22 


B23 


B21 


B25 


B26 








A33 





A35 


A36 


, h2 = 








B33 





B35 


B36 













A45 


Am 











Bu 


B45 


B46 














A55 


^56 
















B55 


B56 

















Am. 



















Bee. 



' A22 


A23 




B22 


B23 





A33_ 







B33 



Theorem 7.2. LeA S G 0{ru,rw,a-,b). Let hi G Pu{ru-,ru-\-a) andh2 G Pw{fWi'^w+ 
a). Then {hi,h2) G Hs if and only if 
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(1) the G{2m — 2ru — 2a) factor in the Levi decomposition of hi is contained in a 
certain G{b) x G{2m — 2r^ — 2a — b); 

(2) the G{2n — 2m — 2ry/ — 2a) factor in the Levi decomposition of /i2 is contained 
in a certain G{b) x G{2n — 2m — 2ry/ — 2a — b); 



(3) the submatrices 



of hi and 



B22 B23 

533 



A22 A23 
A33 

In particular, dim 0{ru, rw, ci,b) = dim H — dim Hs, and 
1 .. .1 -2^-2 

2 



o//i2 in (1351) are identical. 



dim if t 



dimG'(2m) + - dimG'(2n - 2m) + + _ 



2 

1 1 
- - dim G(2m - 2ru - 2a) - - dim G{2n - 2m - 2rw - 2a) + dim G{b) 

(36) + dimG'(2m-2rc7 -2a-6) + dimG(2n-2m-2rvF -2a-6). 

Proof. The preceding discussion has shown that every (/ii,/i2) G Hs must satisfy 
Theorem 17.21 (1)(2)(3). Conversely, if (/ii,/i2) £ Pu{ru,ru + a) x Pw{rw,rw + a) 
satisfies Theorem 17.21 (1)(2)(3). then (/?.i,/i2) preserves 5* by a direct computation. 

It remains to compute dim Hs- Notice that 
dim Pjj {rij , rjj + a) = - [dimG(2m) + rfj + + dimG(2m — 2ru — 2a)] , 



dim Py/ {rw , rw + a) 



[dimG(2n - 2m) +rw + + dimG(2n - 2m - 2rw - 2a)] . 



Taking into the consideration of Theorem 17.21 (1)(2)(3), we have 

dim Hs = dim Pu{ru, ru + a) + dim Pw{rw,rw + 0) 

- dimG(2m - 2ru - 2a) - dimG(2n - 2m - 2rw - 2a) 
+ dim G{2m - 2ru - 2a - b) + dimG{2n - 2m - 2rw - 2a - 6) 
+ dim G{b) — a^ — ab. 

(I36|) then follows. □ 



(37) 



Lemma 7.3. Suppose that ru + rw + a + b = r and inequalities (|33|) hold. If 

0{ru^rw,CL,b) is open in GrGr(r) then 



arjj = 0, 



arw = 0, 



rurw = 0, 



a < 1 



Proof. 0{ru, rw, a, b) is open, if and only if 0{ru, rw, a, b) is of maximal dimension, 
if and only if Hs is of minimal dimension. 

Suppose that aru 7^ 0. Then a > 1 and > 1. Notice 

6+2 6+2 
a — l+ru — l+rw + b+2 = r, r^/ — 1+a — IH < rn, rj^ + a — IH < n—m. 
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Let S' G 0{ru - l,rw, a - l,b + 2). By ([36]), we compute by MAPLE that 

b 

dim(ifs) — dim(if5") = ru + 2{n — m — rp^/ — 1 — -) > 0. 
We see that Hs is not of minimal dimension. 

Similarly if aryi/ 7^ 0, then is not of minimal dimension. Now suppose that 
'^u°''w 7^ 0. Then r^/ > 1 and ryi/ > 1. Notice that 

6+2 6+2 
r{/-l + rvy-l + a + + 2 = r, r^z-l + aH — <'m, r^^-l + aH ^ < n-m. 

Let S' G C(rt/ - 1, rvt/ - 1, a, 6 + 2). By ([3S]), we have 

dim(ifs) — dim(ifs/) = {ru + r^^ — 1) + 2{n — ru — rw — a — b) > 0. 
So if^ is not of minimal dimension. 

Suppose that a > 2. Then 

6+2 6+2 
ru + a — 2-\ — < m, rw + a — 2 -\ — <n — m, ru + rw + a — 2 + b + 2 = r. 

Let S' G 0{ru, rw,a-2,b + 2). By ([36]), we have 

dim(if5) - dim{Hs') = 2a - 3 > 0. 

So Hs is not of minimal dimension. □ 

Lemma 7.4. If 0{ru,rw,a,b) is an open orbit, then 6 must be even and maximal. 

Proof. We have shown in Theorem 17.11 that 6 is even. Lemma 17.31 indicates that at 
most one of ru, rw, cl is nonzero and a < 1. If on the contrary, 6 is not maximal, then 
ru > 2 or rw > 2. Let us say ru > 2. Then rw = a = and S G 0{ru, 0, 0, 6). There 
are 3 possibilities for 6 not maximal: 

(1) There exists an if-orbit 0{ru — 2x, 0, 0, 6 + 2x) for some x G Z"*". Let S' G 
0{ru - 2x, 0, 0, 6 + 2x). Then n - m > i(6 + 2x) = | and 

dim Hs — dim Hs' = 4x(n — m — -) + 2x{ru — 2x) > 0. 

It contradicts the assumption that S is in an open if -orb it. 

(2) There exists an if-orbit O{0, ru — 2x, 0, 6 + 2x) for some x G Z+. Then fl33l) 
implies that 0{ru — 2x, 0, 0, 6 + 2x) is also an if-orbit. Again we reach a 
contradiction. 

(3) There exists an if-orbit O{0,0,l,ru + 6 — 1) (where ru > I is odd). Let 
S' G C»(0, 0, 1, + 6 - 1). Then n - m > |(rc/ + 6 - 1) > | and 

6 

dim Hs — dim Hs' = 2ru{n — m ) > 0. 

2 

This contradicts that S is in an open if-orbit. 
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Therefore, b must be maximal in an open ff-orbit. 



□ 



Theorem 7.5. Let {G,H) = (G(2n),G(2m) x G{2n-2m)). 

(1) When r > min(2m, 2n — 2m), the unique open H -orbit in Gicir) is 



Proof. By Lemma [7l3] and Lemma [7^ it remains to prove that if O{0, 0, 1, r — 1) is an 
if-orbit — in which r < min(2m, 2n — 2m) by (l33l) and r is odd — then its dimension 
is greater than those of 0, 0, r - 1) and 0(0, 1, 0, r - 1). Let S e 0, 0, r - 1) 
and S' e 0(0, 0, 1, r - 1). Then by ([3S]), 

dim Hs - dim Hs' = {2n - 2m - r) + 1 > 0. 

So dimO(0,0, l,r - 1) > dimO(l,0,0,r - 1). Likewise, dimO(0,0, l,r - 1) > 
dimO(0,l,0,r- 1). □ 

Example 7.6. LetG = G{Am + ^) and H = G{2m) x G{2m + 8). The open H- orbits 
in Gicir) for increasing r are: 

0(0,0,1,0), 0(0,0,0,2), 0(0,0,1,2), 0(0,0,0,4), 0(0, 0, 1, 2m - 2), 0(0, 0,0, 2m), 



Theorem 7.7. Theorems \ \7.S\ and \ 7. 5\ hold for all algebraically closed field of 
characteristics not equal to 2. 
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